Devices, systems and methods for natural feature tracking of surgical tools and other objects

ABSTRACT

Systems, methods and devices for use in tracking are described, using optical modalities to detect spatial attributes or natural features of objects, such as, tools and patient anatomy. Spatial attributes or natural features may be known or may be detected by the tracking system. The system, methods and devices can further be used to verify a calibration of a tool either by a computing unit or by a user. Further, the disclosure relates to detection of spatial attributes, including depth information, of the anatomy for purposes of registration or to create a 3D surface profile of the anatomy.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/901,730, (the '730 application) filed on Jun. 15, 2020, the entirecontents of which are incorporated herein by reference. The '730application is a continuation of U.S. application Ser. No. 15/522,559,(the '559 application) filed under § 371(c)(1) on Apr. 27, 2017 from PCTNo. PCT/CA2015/000560 filed Oct. 29, 2015. The entire contents of the'559 application are incorporated herein by reference. The '559application claimed priority to U.S. provisional application No.62/072,041 titled “Systems, methods and devices for anatomicalregistration and surgical localization” and filed on Oct. 29, 2014, theentire contents of which are incorporated herein by reference.

The '559 application claimed priority to U.S. provisional applicationNo. 62/072,030 titled “Devices including a surgical navigation cameraand systems and methods for surgical navigation” and filed on Oct. 29,2014, the entire contents of which are incorporated herein by reference.

The '559 application claimed priority to U.S. provisional applicationNo. 62/084,891 titled “Devices, systems and methods for natural featuretracking of surgical tools and other objects” and filed on Nov. 26,2014, the entire contents of which are incorporated herein by reference.

The '559 application claimed priority to U.S. provisional applicationNo. 62/072,032 titled “Devices, systems and methods for reamer guidanceand cup seating” and filed on Oct. 29, 2014, the entire contents ofwhich are incorporated herein by reference.

FIELD

The present disclosure relates to systems, methods, and devices fortracking of features of tools by an optical tracking system for thepurpose of calibration of tools, verification of the calibration andregistration of an anatomy.

BACKGROUND

In many types of surgery, systems and devices are used to provide asurgeon with real-time positional guidance to guide therapy. Forexample, in THA, there exist systems that provide a surgeon withpositional guidance of an acetabular implant with respect to a pelvis ofa patient. Positional measurements are measured using tracking systems,typically utilizing optical, electro-magnetic, inertial, ultrasonic orRF measurement modalities.

Optical tracking systems, such as the Polaris™ system, manufactured byNorthern Digital Inc. (Waterloo, ON), utilize fixed multi-camera arrayslocated in an operating room to detect targets within a working volumeof the cameras. The targets normally have optically identifiable markersthat are easily identifiable in a video feed of the cameras. Examples ofmarkers include active or reflective markers in the infra-red frequencyspectrum, markers of a distinct colour, markers with distinct shapes orpatterns, which are easily positively identifiable in an image (e.g.checker pattern) etc. Commonly, reflective spheres are used as markers,since their centroids are well-defined and accurate; furthermore,spheres can be viewed from multiple angles. The markers and the camerasmay have matched frequencies to filter unwanted light, either inhardware or in software.

Targets are normally comprised of a plurality of markers. A singlemarker provides an identifiable feature to the optical tracking systemin the form of positional information. A plurality of markers associatedwith a single target would allow the pose (position and orientation) inup to 6 degrees of freedom (DOF) to be calculated by a computing orprocessing unit embedded within, or in communication with the opticaltracking system. Each marker provides a well-defined and accuratefeature. For medical and/or surgical applications, optical trackingsystems are generally used to determine the pose of targets and theobjects to which targets are attached (e.g. surgical instruments ortools, a patient's anatomy, etc.). Targets (comprised of a plurality ofmarkers) must be registered (also referred to as calibrated) to theinstrument to which they are attached; furthermore, they must maintainthe registration throughout use.

BRIEF SUMMARY

There is disclosed a system comprising: a sensor comprising an opticalsensor configured for attachment to a tool at a known positionalrelationship, the tool having an effector with the tool lying within afield of view of the optical sensor, the sensor configured to generatean optical sensor image of the tool and to generate optical measurementsof a target, the target lying in the field of view of the opticalsensor; and a computing unit in communication with the sensor. Thecomputing unit is configured to: calculate a pose of the target in up tosix degrees of freedom using the optical measurements; determine anexpected location of the effector of the tool based on pre-loadedinformation of the tool; determine a location of the effector of thetool based on features of the tool detected from the optical sensorimage of the tool; calculate a difference between the location andexpected location of the effector of the tool; generate a confidencemetric using the difference; and provide positional measurements of theeffector of the tool with respect to the target, the positionalmeasurements being provided with the confidence metric. The computingunit is further configured to: provide the confidence metric to adisplay unit to display in one of a numerical or graphical format; andprevent surgical navigation using the positional measurements when theconfidence metric is outside of a tolerance range. The computing unitdetermines the location of the effector of the tool by: calculating apose of the effector of the tool and further determining the expectedlocation of the effector of the tool by calculating an expected pose ofthe effector of the tool; and by calculating a position of the effectorof the tool within a coordinate frame of the two-dimensional opticalsensor image and further determining the expected location of theeffector of the tool by calculating an expected position of the effectorof the tool within the coordinate frame of the two-dimensional opticalsensor image. The target is configured to: attach to an object; attachto an anatomy of a patient; and provide positional information to theoptical sensor. The tool contains features that comprise opticallydetectable markers. The sensor further comprises a kinematic mount tokinematically couple to a cooperating kinematic mount on the tool, andthe pre-loaded information comprises a first positional relationshipbetween the optical sensor and the kinematic mount of the sensor, and asecond positional relationship between the cooperating kinematic mountand the tool.

There is disclosed a system to provide surgical navigation of aneffector tool with respect to the pose of a target. The systemcomprises: a sensor comprising an optical sensor configured forattachment to the tool at a known positional relationship, the toolhaving an effector with the tool lying within a field of view of theoptical sensor, the sensor configured to generate an optical sensorimage of the tool and to generate optical measurements of the target,the target lying in the field of view of the optical sensor; and acomputing unit in communication with the sensor. The computing unit isconfigured to: calculate the pose of the target in up to six degrees offreedom using the optical measurements; determine an expected locationof the tool based on pre-loaded information of the tool; generate avirtual tool projection based on the expected location of the tool;generate a composite image comprising the optical sensor image, thevirtual tool projection and virtual error bounds; and provide thecomposite image to a display unit. The system further comprises: adisplay unit to display the composite image; and a target configured toattach to an anatomy of a patient and provide positional information tothe optical sensor wherein the computing unit is configured to providesurgical navigation with respect to the target. The sensor furthercomprises a kinematic mount for attachment to a cooperating kinematicmount on the tool, and the pre-loaded information comprises a firstpositional relationship between the optical sensor and the kinematicmount of the sensor, and a second positional relationship between thecooperating kinematic mount and the tool. There is an opticallydetectable marker attached to the tool, and the computing unit isfurther configured to determine the expected location of the tool basedon the pre-loaded information of the tool, the pre-loaded informationcomprising a spatial relationship between the sensor and the opticallydetectable marker, and the virtual tool projection comprising a virtualprojection of the optically detectable marker.

There is disclosed a computer-implemented method capable of:calculating, by at least one computing unit, a pose of a target in up tosix degrees of freedom using optical measurements, generated by a sensorcomprising an optical sensor, the sensor in communication with thecomputing unit, the target lying in a field of view of the sensor;determining, by the at least one computing unit, an expected location ofan effector of a tool, with the tool lying within the field of view ofthe sensor, based on pre-loaded information of the tool, the sensorconfigured to attach to the tool at a known positional relationship;determining, by the at least one computing unit, a location of theeffector of the tool based on features detected from an optical sensorimage of the tool generated by the optical sensor; calculating, by theat least one computing unit, a difference between the location and theexpected location of the effector of the tool; generating, by the atleast one computing unit, a confidence metric using the difference; andproviding positional measurements of the effector of the tool withrespect to the target, the positional measurements being provided withthe confidence metric.

There is disclosed a computer-implemented method to provide surgicalnavigation of an effector of a tool with respect to a pose of a targetby: calculating, by at least one computing unit, the pose of the targetin up to six degrees of freedom using optical measurements generated bya sensor comprising an optical sensor, the sensor in communication withthe computing unit, and the target lying in a field of view of thesensor; determining, by the at least one computing unit, an expectedlocation of the effector of the tool, with the tool lying within thefield of view of the sensor, based on pre-loaded information of thetool, the sensor configured to attach to the tool at a known positionalrelationship; generating, by the at least one computing unit, a virtualtool projection based on the expected location of the effector of thetool; generating, by the at least one computing unit, a composite imagecomprising an optical sensor image generated by the optical sensor andthe virtual tool projection; and providing the composite image to adisplay unit.

There is disclosed a system comprising: a sensor comprising an opticalsensor configured to generate optical measurements of a target andsimultaneously generate an optical sensor image of an anatomy of apatient; a target configured to be attached to the anatomy; and acomputing unit in communication with the sensor. The computing unitconfigured to: calculate a pose of the target attached to the anatomyusing the optical measurements; measure spatial attributes of theanatomy using the optical sensor image; and determine a registration forthe anatomy based on the pose of the target and the spatial attributesof the anatomy. The sensor is further comprised of a depth sensor togenerate a depth image, wherein the depth sensor is: one of a time offlight camera, a laser scanner, and a camera with illuminatingcomponents; and in a known and fixed relationship with respect to theoptical sensor. The computing unit is further configured to: measure thespatial attributes of the anatomy additionally using the depth image;determine a 3D surface profile of the anatomy based on the spatialattributes; determine the registration of the anatomy using a pluralityof poses of the target and optical sensor images from a plurality ofvantage points; determine the 3D surface profile of the anatomy using aplurality of poses of the target and depth images from a plurality ofvantage points; and use a digital 3D scan of the anatomy and acorrespondence between the optical sensor image and the digital 3D scanto determine the registration for the anatomy.

There is disclosed a sensor, in communication with a computing unit, toprovide surgical navigation. The sensor comprises: a first opticalsensor configured to generate optical measurements of a target and witha first field of view; and a second optical sensor located at a knownand fixed positional relationship from the first optical sensor, andfurther configured to generate optical measurements of features of anobject and with a second field of view, the second field of viewoverlapping with the first field of view. The first optical sensor andthe second optical sensor share at least one optical component, theoptical component being one of an imager and a lens. The first opticalsensor and the second optical sensor are a single optical sensor. Thesensor further comprises a depth sensor, the depth sensor configured to:be positioned in a known and fixed relationship with respect to thefirst optical sensor and the second optical sensor; and have anoverlapping field of view with the first field of view and the secondfield of view. The sensor is configured for attachment to an anatomy ofa patient and further comprises a kinematic mount for mounting to acooperating kinematic mount.

There is disclosed a system to determine object poses during a surgicalnavigation procedure, the system comprising: a sensor unit comprising anoptical sensor configured to generate images of objects within a fieldof view of the optical sensor; and a computer processing unit, incommunication with the sensor unit and memory storing instructions that,when executed by the processing unit, cause the system to perform amethod comprising: storing first natural feature information of a firstobject, the natural feature information defining information to identifythe first object in the images using natural visual features of thefirst object; receiving from the sensor unit an image of the firstobject; extracting second natural feature information from naturalvisual features of the first object in the image; and calculating thepose of the first object relative to the sensor unit using the firstnatural feature information and the second natural feature informationto facilitate performance of the surgical navigation procedure.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments disclosed herein will be more fully understood from thedetailed description and the corresponding drawings, which form a partof this application, and in which:

FIG. 1 depicts a surgical tracking system with a sensor for use in asterile field, with a target in a field of view of the sensor;

FIG. 2 illustrates an impactor tool highlighting its geometricalfeatures as an example for clarity;

FIG. 3 a depicts a surgical tracking system with a sensor to track atool without a target, and using only features of the tool, inaccordance with an embodiment;

FIG. 3 b depicts an image of a field of view of the optical sensor, whenthe sensor is pointed at the tool (as shown in FIG. 3 a );

FIG. 4 illustrates a curved impactor tool highlighting its geometricalfeatures as an example for clarity;

FIG. 5 illustrates a registration between a target attached to a tool,and a feature of the tool, as an example for clarity;

FIG. 6 depicts a photograph of an actual optical sensor image whenviewing an impactor tool with a target and an implant attached to it;

FIG. 7 a illustrates a sensor pointing at a tool with a tip, and atarget;

FIG. 7 b is a line drawing showing a representative optical sensor imageof the sensor pointing at the tool with a tip, and a target;

FIG. 8 a illustrates a sensor pointing at a tool with a bent tip, and atarget;

FIG. 8 b is a line drawing showing a representative optical sensor imageof the sensor pointing at the tool with a bent tip, and a target;

FIG. 9 is a line drawing showing a representative optical sensor imageof the sensor pointing at the tool with a bent tip, a virtual projectionof where the tip should be, and a target attached to the tool;

FIG. 10 depicts, as an example for clarity, virtual error boundsprojected on a composite image of a tip of a tool;

FIG. 11 a shows a sensor kinematically coupled to a probe and a targetattached to a patient's body at an anatomical feature of interest as anexample for clarity;

FIG. 11 b is a line drawing showing a representative optical sensorimage depicting a feature to verify calibration between the opticalsensor and the tool;

FIG. 12 a shows a sensor kinematically coupled to a bone cuttinginstrument with an optically identifiable marker and a target attachedto a patient's body as an example for clarity;

FIG. 12 b is a line drawing showing a representative optical sensorimage depicting a feature to verify calibration using an opticallytrackable marker;

FIG. 13 shows a sensor kinematically coupled to a robotic manipulatorand a target attached to a patient's anatomy as an example for clarity;

FIG. 14 shows a sensor coupled to a calibration tool for use with animpactor as an example for clarity;

FIG. 15 shows use of a surgical tracking system to register anatomy to atarget in accordance with an embodiment;

FIG. 16 a illustrates a sensor detecting spatial attributes of a face tocreate a registration to a target, in accordance with an embodiment;

FIG. 16 b is a line drawing showing a representative optical sensorimage of a sensor viewing spatial attributes of a face, in accordancewith an embodiment;

FIG. 17 depicts a 3D scan of a patient's anatomy that can be used inregistration;

FIG. 18 a and FIG. 18 b depict line drawings showing representativeoptical sensor images from two vantage points;

FIG. 19 shows a system with a target on a pelvis, and a depth sensorscanning an acetabular surface;

FIG. 20 shows a system with a target on the depth sensor scanning anacetabular surface, and an optical sensor on a pelvis;

FIG. 21 shows a hardware configuration of a sensor with a switchableoptical filter for use in different spectra; and

FIG. 22 shows a hardware configuration of an optical sensor using twoimagers.

FIGS. 23 and 24 show flowcharts of respective methods, in accordancewith embodiments.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity.

DETAILED DESCRIPTION Optical Tracking Systems—Introduction

Several systems, methods and devices will be described below asembodiments. The scope of the claims should not be limited by theembodiments set forth in the examples, but should be given the broadestinterpretation consistent with the description as a whole.

Reference in the specification to “one embodiment,” “preferredembodiment,” “an embodiment,” or “embodiments” means that a particularfeature, structure, characteristic, or function described in connectionwith the embodiment is included in at least one embodiment, and may bein more than one embodiment. Also, such phrases in various places in thespecification are not necessarily all referring to the same embodimentor embodiments.

An optical surgical tracking system has been described in previouslyfiled patent documents (for example, see applications U.S. 20120157887titled “Method and system for aligning a prosthesis during surgery usingactive sensors”, the entire contents of which are incorporated hereinand U.S. 20140275940 titled “System and method for intra-operative legposition measurement”, the entire contents of which are incorporatedherein). An exemplary system applied to Total Hip Arthroplasty (THA) maybe seen in FIG. 1 . This optical tracking system 100 comprises a sensor102, further comprising an optical sensor (such as, a camera) with afield of view (FOV) 104. The sensor 102 may be attached to objects 106such as, a patient's bone and surgical tools and instruments. The systemalso comprises one or more targets 108. The targets 108 can be trackedby the optical sensor when the target 108 lies in the FOV 104 of theoptical sensor to calculate pose (position and orientation), and canalso be attached to various objects, including patient anatomy, toolsand instruments. The sensor 102 is in communication with a computing orprocessing unit 110 via a cable 112. The system may include a displayunit 114 to display measurements obtained from the pose of the targets108 to a surgeon. In this example, an acetabular implant 116 is shownattached to its insertion tool 118, referred to as an acetabularimpactor. The target 108 is attached to a femur bone 120, and the sensor102 is attached to a pelvis 122.

This present disclosure uses THA as an illustrative example; however, itmust be appreciated that the present disclosure extends to all medicalinterventions where real-time positional measurements of objects arevaluable. Objects 106 are intended to be surgical tools or instruments,or parts of the anatomy of a patient. Furthermore, in some of thepresent examples, the sensor 102 will be shown as attached to apatient's pelvis 122 located in a surgical sterile field. It must beappreciated that this disclosure is not limited to attaching the sensor102 to the bone of a patient; however, there may be advantages in doingso, such as, the inherent close proximity between the optical sensor andthe objects being tracked.

In this specification, the sensor 102 is used for measurement of posesof targets 108 attached to objects 106 that are within its field of view104. It is not a medical imaging or visualization device, such as anultrasound probe or an endoscope. The optical sensor requires an opticalcalibration, as well as a rigid construction to maintain thecalibration, to enable precise pose measurements of a target.

Optical Tracking Based on Natural Features of an Object

Prior art discloses targets, comprising a plurality of opticallydetectable markers, that are attached to objects, including tools,instruments and anatomy in navigated surgical procedures. As illustratedin FIG. 2 , the object 106 may be an acetabular impactor 202. Instead ofusing a target 108 to track the impactor 202, the natural features ofthe impactor may be used to track it. Natural feature tracking (NFT)aims at determining the pose of an object using an optical sensor toidentify natural features associated with the object (natural featuresbeing inherent spatial features of the object itself, as opposed to atarget, whose fundamental purpose is to provide positional signals forcalculation of positional measurements). The natural features areprocessed by a computing unit 110, and pose measurements are computed.For example, an impactor 202 may have known features, which could beidentified in a two-dimensional optical sensor image, and used todetermine the pose of the impactor in up to 6 DOF using the naturalfeatures. Impactors often have a straight cylindrical shaft 204 alongthe same axis as the acetabular implant; the acetabular implant 206itself is hemispherical, such that it has an opening plane as well as acentroid; impactors may have other features, such as changes in shape ordiameter; sections of an impactor may be textured or coloured 208;sections of an impactor may be of known dimensions 210 (e.g. length).All of these exemplary natural features have a particular spatialrelationship to the overall pose of the impactor.

In many applications, a 6 DOF pose of an object is not required toprovide clinically useful measurements. For example, since theacetabular implant 206 is hemispherical, its orientation about the axisof its opening plane is not important. Therefore, when aligning theacetabular cup 206, only 2 DOF in orientation are useful. Since in THA,the angle of the cup 206 as it is placed inside the pelvis 122 isimportant (and not necessarily the translational position), a relevantpose measurement would include only 2 DOF. In the context of thisspecification, tracking the natural features of the object 106 mayinclude the determination of a pose of an object 106 in up to sixdegrees of freedom.

FIG. 3 a and FIG. 3 b illustrate an example of THA during which NFT isapplied to the step of guiding the insertion of an acetabular cup 206using an impactor tool 202. An impactor 202 is within a FOV 104 of anoptical sensor attached to a pelvis 122 as the surgeon is positioningthe impactor 202. The two-dimensional optical sensor image 302 as seenby the optical sensor is shown in FIG. 3 b . In this figure, the opticalsensor image 302 of the impactor is depicted. The image 302 containsmany features that may be extracted using image processing operations.For example, the straight edges 304 of the cylindrical shaft 204 may bedetected using edge detection operations. Due to the perspective effect,the edge lines of the shaft 204 will not be parallel, but rather angledtowards a vanishing point; the two lines as projected onto the image, aswell as the angle between them, can be used to calculate the 2 DOForientation of the impactor (in real time). These measurements can bedisplayed to a surgeon who is using the system 100 to accurately placethe implant 206 within a patient's pelvis 122 at a particular desiredangle. The edges of the straight cylinder are exemplary features; anyother identifiable feature may be used.

Due to the close proximity of the sensor to the surgical objects beingtracked, NFT may be used favourably in the optical tracking systemdescribed herein. In a traditional optical tracking system disclosed byprior art, where the cameras are located outside a sterile field of asurgery, the working volume of the cameras is quite large, and beingable to accurately and positively track a surgical object using NFT isnot feasible. Conversely, when the camera is in close proximity to thetracked objects, the tracked objects provide more information within thecamera image, which can be used for accurate and repeatablelocalization. To aid in the identification of objects within the image,the sensor may comprise illuminating components, such as, infra-red LEDsto illuminate a scene containing the object of interest, thus enhancingthe information content within the image by which the optical sensor isable to accurately track the object via its natural features.Furthermore, where the optical sensor is in close proximity to thetarget (e.g. when it is attached to the patient's anatomy), furtherbenefits arise, such as, a more pronounced visual perspective effect,which may aid in the pose calculation using NFT.

NFT Based on a Priori Feature Information

To calculate the pose of an object based on NFT, information about thefeatures of the object may be known a priori. For simple objects, suchas a straight impactor (or for example, a biopsy needle), the a prioriinformation may be that the tool is a straight shaft. This informationcan be used by the image processing and pose solving operations(executing on a computing unit) to calculate pose of the object. In somecases, objects may have complex feature definitions. For example, asillustrated in FIG. 4 , in THA, a curved impactor 402 may be used.Complex shapes of the object may require more complex a priori featureinformation to be available to the image processing and pose solvingoperations. To address this issue, pre-loaded feature information may beaccessed by the computer operations, such information being specific tothe object or class of objects being tracked. The pre-loaded featureinformation may be in the form of electronic Computer-Aided Design (CAD)files, stored in memory on the computer. A database of CAD files forvarious tools or objects may be stored in memory (or accessed via anetwork) to provide the image processing and pose solving operationswith this information. The pre-loaded information may include thegeometry or shape of the object. Where the object is a tool, and thetool has an effector, pre-loaded information may include the positionalrelationship of the effector of the tool with respect to other parts ofthe tool. An effector of a tool is a part of the tool that has thegreatest effect in achieving its purpose, e.g.: blade of a scalpel, tipof a probe, etc.

The operations use the pre-loaded information to assist in determiningthe pose of the object. For example, the operations, executing on acomputing unit, may receive a raw image from the optical sensor (opticalsensor image), process the optical sensor image to identify features tobe used as an input, load a priori feature information of the object,perform optimization operations to match identified input features witha priori feature information and if a suitable match exists, calculatethe pose accordingly.

In some instances, the a priori feature information may not beavailable. In such a case, the feature information may be generated inadvance of the real-time pose measurements, by “teaching” the system thefeatures of the object. For example, multiple views of an object (e.g.impactor tool) may be captured through the optical sensor from differentvantage points. Key features (e.g. shaft axis) may be identifiedautomatically (i.e. by the computing unit), semi-automatically (i.e. bythe computing unit with limited user intervention) or manually (i.e. bythe user, for example, identifying specific features by clicking on themon a computer screen) and saved in software (e.g. in computer memory).These features may now be accessed for use in pose solving operations asdescribed above.

Natural Feature and Target Tracking (Serial)

In some instances, it may be appropriate for the system 100 to trackboth natural features of an object and targets attached to the same ordifferent objects at different steps during navigated surgery. Forexample, in THA, NFT may be useful for angular measurements used inplacing an acetabular cup 206 in a pelvis 122, but not for measuringchanges in leg position. This is because a femur 120 may not possessclearly defined features that can be detected by the optical sensorand/or processed by operations for pose calculation. Whereas an impactorshaft 204 is a well-defined and unambiguous shape, a femur 120 isirregularly shaped, covered in soft tissues and blood, and varies frompatient to patient. Therefore, it is appropriate to track a patient'sleg position in THA using a target 108, while it remains advantageous totrack an impactor 202 without a target, for simplicity, ease of use andaccuracy. In reference to FIG. 1 , a femur 120 is tracked with a target108, whereas an impactor 202 is depicted without a target. There aremany examples outside of THA in which it is appropriate to track certainobjects with a target, and others via NFT, during the same procedure,using the same system.

In order to track both targets and natural features using the samesystem, the system is capable of detecting a target and the naturalfeatures of an object. Either both can be detected within the sameoptical spectrum (e.g. visible, infra-red, etc.) or the optical sensorcan detect the target and the natural features separately (e.g. indifferent spectra—infra-red spectrum for markers on a target, visiblespectrum for natural features), but relative to the same coordinatebaseline reference of the optical sensor. Further detail on this pointis included below.

NFT of an Object Using its Natural Features and a Target

In one embodiment, the system 100 tracks natural features and targetssimultaneously. The natural features may be of an object to which thetarget 108 is coupled. For example, the system may provide functionalityto calibrate an impactor tool, in which a positional relationship 502between an implant (e.g. acetabular cup) and a target is to bedetermined. FIG. 5 illustrates an impactor tool 202 with an attachedacetabular implant 206 and a target 108 attached thereto. This tool 202is used when inserting the acetabular implant 206 during THA. In orderto calibrate the tool 202, the sensor 102 may track the target 108, andsimultaneously track natural features of the acetabular implant 206.

FIG. 6 shows an image 302 from a video feed of the sensor 102 in which atarget 108 and an implant 206 are both attached to an impactor 202 andare visible to the sensor. A step of calibration to determine thepositional relationship 502 between the target 108 and the implant 206may be performed by tracking the target 108, as well as trackingfeatures on the implant 206 such as, its centroid, diameter, openingplane, etc. In this example, the implant 206 is the effector of theimpactor 202. Calibrating this object involves determining the relativepose of the implant and the target, which may be computed bysimultaneously calculating the pose of the target and implantindividually, and then determining a spatial transformation to relatetheir respective coordinate systems. In one embodiment, multiple viewsof the target and implant from different vantage points may reduce thecomputational complexity of the pose solving operations that areexecuted to determine the positional relationship 502 between theimplant 206 and the target 108. This scenario may arise for severalreasons, for example: it may be difficult to determine the pose of theobject depending on the angle from which the natural features are beingviewed; it may be difficult to determine the pose of the object based onits geometry (e.g. determining the orientation of a substantiallyspherical object); there may be sparse a priori information availableabout the natural feature geometry. In such cases, multiple views frommultiple vantage points of the sensor may help provide more informationcontent about the natural features of the effector itself, as well asthe spatial relationship between the effector and the target. It will beappreciated by those skilled in the art that this increased informationcontent is helpful in accurately measuring the pose of the effector, aswell as the relative pose of the effector and target.

Target on Object: Verification of Calibrated Positional Relationship

In addition to calibrating objects prior to use, simultaneous trackingof the object and the target can provide a run-time accuracy check toensure that the object is still calibrated, based on pre-loadedinformation representing the expected spatial relationship between thetarget and the object (e.g. tool). Pre-loaded information may be madeavailable to a computing unit any time prior to verification of thecalibration. The pre-loaded information may be based on a previouslyperformed calibration, manufacturing tolerances and geometry, generatedfeature information through the previously-described learning process,etc. For example, the target may be used for pose calculation of theobject (in this case, an instrument), and the natural features may betracked redundantly to provide a confidence metric that the instrumentremains calibrated. Using a navigated biopsy as an exemplary surgicalprocedure, FIG. 7 a illustrates a sensor 102 with a field of view 104within which there is a biopsy needle 702 and its tip 704. There is atarget 108 attached to a handle 706 of the needle 702. The tip 704 ofthe needle 702 itself may be tracked to ensure that it remains correctlypositioned with respect to the target 108. FIG. 7 b shows atwo-dimensional image 302 as seen by the optical sensor with the tip 704in a calibrated position with respect to the target 108. The sensor 102is able to detect the target 108 in the scene, as well as detect alocation of the effector 704 of the needle 702 based on its naturalfeatures (e.g. in this case, the effector being the tip 704—the endpoint of a long straight shaft). The natural feature information may beavailable a priori or be taught to the system 100, as described inprevious embodiments. The detected tip position is the location of theeffector. A computing unit in communication with the sensor computes apose of the target 108, and determines a calculated tip position (i.e.expected position where the tip is expected to be). The calculated tipposition relies on a priori knowledge of where the tip is expected to bespatially with respect to the target (in general, the tool may attach tothe target according to a known positional relationship). This a prioriknowledge may be from a previously-performed calibration, or based onexpected spatial relationships from manufacturing tolerances of thetarget and/or tool. The a priori knowledge is preferably accessible tothe computing unit as pre-loaded information. The calculated tipposition may be compared with the detected tip position to calculate adifference, if there is one. This difference is used to generate aconfidence metric. The comparison may be performed by the computingunit. After comparison, if the confidence metric is within a tolerancerange defined for a particular application of the system, the system 100may display results of the verification of calibration or displaymeasurements accordingly in a numerical or graphical format, asdescribed below. Furthermore, the computing unit 110 may alert the userthat the needle 702 is out of calibration such that the user can takeappropriate steps (e.g. discard needle, recalibrate needle, etc.), alsodescribed below.

The computing unit may provide a confidence metric based on thecalibration verification results. The confidence metric may represent aconfidence of the accuracy of the positional localization of theeffector of the tool. The confidence metric may be provided to a displayunit for display to a user based on which the user may adjust theirusage of the system. For example, the user may choose to perform are-calibration or discontinue use of the system if the localizationconfidence is low). The confidence metric may be displayed graphically,numerically, or in any other suitable fashion.

Furthermore, calibration verification results and/or a confidence metricmay be provided for further processing such that the surgical navigation(e.g. navigation and/or robotic surgery) system takes appropriateaction. For example, in the event of compromised calibration (e.g. lowconfidence metric), the computing unit may discontinue providingpositional measurements to a display unit (in this way, the user wouldbe prevented to seeing potentially inaccurate positional measurementsrelated to the effector). In another example, in a robotic surgeryapplication, a compromised calibration may cause the processing unit tosend instructions to prevent the robot from proceeding with its plannedtrajectory (which would potentially cause harm to a patient, as therobot would be relying on inaccurate positional measurements related tothe effector).

For further clarity, FIG. 8 a illustrates a sensor 102 with a field ofview 104 within which there is a biopsy needle 702 that is bent with itstip 704 in a location that is different from its expected location. Inaddition, there is a target 108 attached to a handle. FIG. 8 b depicts atwo-dimensional image 302 as seen by the optical sensor. As depicted,the actual tip 704 of the needle 702 is not where it is expected to be,with respect to the target 108. The sensor 102 is able to detect thepose of the target 108, as well as detect the tip 704 of the needle 702based on its natural features. The sensor 102 is able to determine acalculated tip position 802 and the expected calibration between thetarget and the tip. The computing unit may perform operations to verifythe calibration of the tool, which would determine whether the tool(i.e. the needle) is out of calibration.

According to this embodiment, the natural features need not be trackedin 6 DOF. The number of DOF that are tracked by a sensor may depend onthe requirements of the application. For example, where the orientationof the shaft of the needle does not matter for the application, only the3 DOF related to a position of the tip of the needle tip may bedesirable. Conversely, where the orientation of the shaft is requiredfor the application, then 5 DOF needle tracking may be desirable: 3 DOFfor position, and 2 DOF for orientation (the 3^(rd) orientation degreeof freedom being rotation about the axis of the shaft, which do not needto be measured since the shaft is symmetrical about this axis).

Furthermore, rather than measure the pose of the object based on itsnatural features in up to 6 DOF in a world coordinate reference frame(WCRF) (i.e. physical space), it may be desirable to measure theposition of the object (e.g. needle) in a coordinate frame of thetwo-dimensional image as seen by the optical sensor in the opticalsensor image frame (OSIF). Note: the WCRF and OSIF are related: objectsin the WCRF are projected as a 2-dimensional optical sensor image,depending on the specifications of the optical sensor, e.g.: type oflens, optical length, etc. The advantage of measuring the object'sposition in the OSIF rather than WCRF, is that it may be simpler tomeasure the object's positional features in the OSIF as fewercalculations are required to be executed (e.g. pose calculation may notbe required).

For example, a method to verify calibration of a tool and its effectormay be implemented by the computing unit and applied to the example ofthe biopsy needle is provided below. The computing unit receives asensor image that includes target and the natural features of the needle(including needle tip), detects the target (attached to the needle),calculate pose of the target in WCRF, based on a priori featureinformation of the needle itself, as well as the pose of the target inWCRF, determine expected pose of the tip of the needle, project theexpected pose of the calculated needle tip into the OSIF, detect theactual position of the tip of the needle in OSIF, and compare calculatedand detected needle tip positions in OSIF and provide the difference.The difference may be displayed on a display unit. The difference may bedisplayed graphically or numerically.

Target on Object: Verification of Calibration by User

In another embodiment for verification of the calibration of a tool, asystem displays to a user (via a display unit) an image as seen by theoptical sensor. This may be done in the form of a video feed of imagesas seen by the optical sensor. With pre-loaded information about thespatial features of a tool and its effector, the system can display tothe user an expected image of the tool in the format of atwo-dimensional image in OSIF, based on the pose of the target attachedthereto. A user is then able to compare the location of the tool withthe expected location of the tool, as depicted in the image of theoptical sensor. In this embodiment, the computing unit does not need toexecute operations to identify natural features in the optical sensorsimage; instead, a user is able to visually identify the object's naturalfeatures. This method may provide a sense of confidence to a user, asthey are able to verify the accuracy of the calibration of a toolthemselves, by visually comparing the actual and expected locations ofthe tool's effector. This embodiment may further involve displaying avirtual tool projection (i.e. a computer generated representation of thetool's features of interest). The virtual tool projection is based onthe pose of the target, and graphically represents where the computingunit expects the tool to be within the image. The virtual toolprojection may be overlaid or superimposed on the images from theoptical sensor and displayed to the user as a composite image, whichalso display the natural features of the tool itself. The user may thenvisually compare the natural features of the tool with the virtual toolprojection, and verify whether they are sufficiently aligned.

For example, with respect to needle navigation, FIG. 9 shows an image302 as seen by the optical sensor including a target 108 attached to theneedle 702. The needle has a tip 704 and an axis. The tip 704 of theneedle 702 is bent. The figure also shows a virtual tool projection 902,based on the pose of the target 108. Because the needle 702 is bent, thevirtual tool projection 902 is not aligned with the actual needle 702. Auser may visually detect that the virtual needle and the actual needleare not in alignment, and proceed accordingly.

A computing unit method associated with this embodiment comprisesreceiving an optical sensor image comprising a target and naturalfeatures of interest of an object, detecting the pose of the target,calculating pose of the target in WCRF, determining pose of object basedon target pose in WCRF, generating a graphical representation (virtualprojection) of object based on object pose projected into OSIF andpre-loaded feature information of the object, and providing displayinformation in the form of a composite image comprising the opticalsensor image with superimposed virtual projection of the object. Theuser is then able to compare object natural features of interest withvirtual object projection to verify calibration.

This embodiment may provide further features to aid a user in assessingthe calibration of the object. For example, as illustrated in FIG. 10 ,the computing unit 110 may further generate virtual error bounds 1002for display. The virtual error bounds 1002 may be used to assess thedegree to which the object (e.g. tool, instrument, etc.) is out ofcalibration, when the object in the optical sensor image 302 is not inalignment with the virtual projection 902. For example, in FIG. 10 , theobject is not in perfect alignment with the virtual projection, but theobject lies within the virtual error bounds. The virtual error bounds1002 are depicted as lines in FIG. 10 ; however, there may be any otherappropriate shape (or visual element) to represent an acceptableaccuracy of the object. For example, the virtual error bound 1002 may bea bounding volume around an implant; a sphere (projected on the image asa circle) around the tip of an instrument; etc. Also, the virtual errorbounds 1002 may be tied to a geometrical tolerance in 3D space (e.g. itmay be important that the tip of a probe be within a 2 mm sphere). Thevirtual error bounds may be applied in 3D space (that is, within theWCRF), but displayed as a 2D projection onto the image (that is, on theOSIF). In this way, the virtual error bounds 1002 are scaled accordingto their desired accuracy tolerance in 3D space.

NFT of Sensor-Mounted Tool+Simultaneous Target Tracking

In one embodiment, natural features of an object coupled to the sensorare detected and used for improved localization with respect to a poseof a target. Reference is now made to FIG. 11 a in which the object is atool. The tool is a probe 1102 that is used to localize an anatomicallandmark or feature 1104, e.g. on a bone, within a body cavity, or alesion within soft tissues in the brain, is attached to the sensor 102.A target 108 is attached to the anatomical feature 1104. The probe 1102may be kinematically coupled with the sensor 102 through a kinematicmount 1106 on the sensor and a cooperating kinematic mount 1108 on theprobe 1102 such that, when in calibration, the sensor 102 and the tip1110 are in a known positional relationship in up to 6 DOF (comprising afirst positional relationship between the sensor 102 and its kinematicmount 1106, and a second positional relationship between the probe 1102and its kinematic mount 1108) with respect to the sensor 102. Moregenerally, where kinematic mounts are not used, the sensor may attach tothe tool according to a known positional relationship (a knownpositional relationship being any location that may be made known to thecomputing unit via the pre-loaded information prior to performing acalibration verification). When the probe 1102, with its tip 1110 is incontact with the anatomical feature 1104, the target 108 is within theFOV 104 of the sensor 102, and a relative pose between the sensor 102and target 108 is captured.

When the sensor is kinematically coupled to the probe and the featuresof interest of the probe are within the field of view of the sensor, thesensor may track the natural features of the probe to verify thecalibration of the positional relationship between the probe and thesensor. All of the previously-described methods for determining the poseof an object with respect to a sensor based on natural feature trackingmay apply. In this embodiment, the sensor and the probe have a constantpositional relationship. For this reason, it is not necessary tosimultaneously track a target and the natural features of the probe.

Where the tool's features of interest are visible to the sensor when thesensor is coupled to the tool, the sensor may track the natural featuresof the tool to verify the positional relationship between the sensor andthe tool. All of the previously-described methods for determining thepose of an object with respect to a sensor based on natural featuretracking apply. For example, the verification may be performed by aprocessing unit or via user-assessment. The parameters to verify thecalibration may be in expressed in the WCRF or in the OSIF. In thisembodiment, the sensor and the instrument/tool have a constantpositional relationship. One purpose of this embodiment is to measurethe pose between a target and the effector of a tool, using the sensorto localize the target. However, while calibrating or verifying thepositional relationship between the tool and the sensor, the target neednot be trackable, or even within the field of view of the sensor. Thisis because the tool and the sensor have a constant/fixed positionalrelationship, and therefore the tool is visible to the sensor (enablingcalibration and/or verification) regardless of whether the target isalso visible to the sensor.

The verification of the calibration between an effector of a tool, whilethe tool is attached to a sensor at a known positional relationship inup to 6 DOF, may be used to detect issues with the system's accuracy(e.g. damage to the tool and/or effector, malfunctioning optical systemetc.). If the sensor 102 is kinematically coupled to the tool,verification of the calibration may be used to check whether thekinematic coupling is accurate. FIG. 11 b illustrates an image 302 asseen by the optical sensor that corresponds to the physicalconfiguration in FIG. 11 a . The optical sensor image 302 includes thetarget 108 within its FOV 104, as well as the tip 1110 of the probe 1102(via its natural features). A calculated probe tip 1114 is shown in theimage 302. The calculated position of the tip of the probe is based onpre-loaded information of the positional relationship between theinstrument (e.g. probe) and the sensor. The calculated probe tip 1114(also called the reference probe tip) represents where the tip is in anaccurate calibration. When the location of the probe tip 1116 differsfrom the expected location of the reference probe tip, the kinematicmount may not being seated properly (if a kinematic coupling existsbetween the sensor and the probe), there may be a mechanical issue suchas, a bent probe tip, or there may be another system inaccuracy. Theability to detect this inaccuracy between the sensor and the tool canprevent incorrect therapy from being delivered to a patient. Forexample, if the probe is intended for a tumor biopsy, an incorrectkinematic relationship would cause the wrong tissue to be biopsied. Theoptical sensor image 302 includes a target image 1118. It is noted thatthe target image 1118 (in FIG. 11 b ) is not required for calibrationand/or verification of the positional relationship between the sensorand the tool (as previously discussed).

A comparison between the actual probe tip 1116 and the calculated probetip 1114 may be performed by a computing unit, and a difference betweenthe actual pose and expected pose of the tip may be generated.Alternatively, the comparison may be performed by a user, where avirtual tool projection is superimposed on the actual optical sensorimage, and displayed to a user as a composite image via a display unitsuch that the user can visually confirm if the virtual (expected) probetip is in sufficient alignment with the actual probe tip. As previouslydescribed, there are multiple methods to depict the inaccuracy to theuser (e.g. virtual error bounds). Furthermore, if the actual toollocation (e.g. probe tip) is measured, and the reference tool location(e.g. reference probe tip) is known, the computing unit software cancompensate for the inaccuracy and display measurements that account forthe inaccuracy (i.e. perform a run-time self-calibration). This may beaccomplished by measuring an error pose between the actual and referencetool locations, and applying the inverse of the error pose to subsequentmeasurements.

Non-NFT Verification of Calibration of Sensor-Mounted Tool

In one embodiment, a non-natural feature may be used to verify thecalibration of the positional relationship between a sensor 102 and anobject. For example, with reference to FIG. 12 a , the object is ahaptic/robotic bone cutting tool 1202 (e.g. burring tool). The tool 1202is configured to cut and/or remove bone 1204 under guidance provided bya relative pose measurement between the sensor 102 and the target 108,when that bone 1204 has a target 108 attached to it. The sensor 102 hasa kinematic mount 1206. The tool 1202 has a cooperating kinematic mount1208 for kinematic coupling to the sensor 102, such that there is aknown positional relationship (in up to 6 DOF) between the sensor 102and the tool 1202. The tool provides an optically trackable feature 1210that lies within the FOV 104 of the sensor 102. The optically trackablefeature 1210 is not a natural feature related to the geometry of thetool 1202; rather, it is a dedicated feature in a known spatialrelationship with the position of the end-effector of the tool, andintended to be optically detected by the sensor.

Reference is made to FIG. 12 b . The optical sensor image shows theoptically trackable retro-reflective marker 1210. In the computing unit,the position of the reference marker is known (i.e. the expectedposition of the marker) based on pre-loaded information. The accuracy ofthe calibration is verified when the positions of the reference markerand actual marker align/overlap. If not, then the computing unit maynotify the user to prevent incorrect therapy from being delivered to thepatient. This comparison may be done by the processing unit or via auser assessment relying on a display of the virtual projection of theoptically trackable feature superimposed on the actual image of theoptically trackable feature as a composite image. Both options may relyon previously-described techniques/methods.

By measuring the position of the marker in the two-dimensional opticalsensor image, the system may be tolerant to a compromised, or unknownpositional relationship between the end-effector and the sensor; acalibration or compensation may be performed based on the expectedposition of the marker in the image. It should be noted that multiplemarkers, or other features visible to the sensor, may be used to detectand/or compensate for a compromised positional relationship between thesensor and the instrument. It should be noted that where roboticactuation is used to assist with delivering guided therapy, verifyingthe accuracy of the end-effector is critical since the robot isfunctioning autonomously and the user/surgeon does not relying on humanclinical judgment.

Clinical Applications

Those skilled in the art will appreciate how the following applicationsmay benefit from the concepts described herein. In one embodiment, withreference to FIG. 13 , the sensor 102 is attached to a tool, which is anend-effector 1302 of a robot manipulator 1304 (e.g. used for hapticallyguided and/or robotic surgery). The robot manipulator 1304 has a basesurface 1306 that is anchored to the ground (i.e. to some referencewithin a room), or to a patient's anatomy (e.g. to a bone, such as arobotic knee cutting guide). The sensor 102 is kinematically coupledwith the end-effector 1302, and the position of the end-effector 1302with respect to a bone 1308 is tracked in real time using the target108.

In one embodiment, with reference to FIG. 14 , a calibration tool 1402is kinematically coupled to the sensor 102. The target 108 is coupled toa surgical instrument that needs calibration (e.g. an acetabular cupimpactor 202 used in THA). The calibration tool 1402 has features suchas, a calibration contact surface 1406 that is configured to mate withthe surface of an opening plane 1408 of the acetabular cup 206. When thecalibration tool contact surface 1406 is co-planar with the cup plane1408, the sensor 102 to target 108 pose is captured, and used to computethe calibration of the surgical instrument (i.e. the positionalrelationship between the target and the plane of the cup).

Natural Feature Scanning+Simultaneous Target Tracking Using OpticalCamera for NFS

In addition to calibrating a target to a tool by simultaneously trackingtargets and natural features, registration of a target to anatomy isalso contemplated. By way of example, in FIG. 15 , a target 108 isattached to a spine 1502 of a patient who is undergoing a spinalsurgical procedure. The sensor 102 is aimed (either manually by thesurgeon, or otherwise) at the target 108, but also has several spatialattributes of the anatomy, in this case exposed vertebrae, within itsFOV 104. The sensor 102 measures the pose of the target 108, andsimultaneously measures the pose of the vertebrae, thus allowing therelative pose between the anatomy (i.e. vertebrae) and the target to bedetermined. This relative pose is referred to as an anatomicalregistration.

An example in accordance with an embodiment illustrated in FIG. 16 a .The target 108 is coupled to a patient's head (invasively ornon-invasively). The sensor 102 captures spatial attributes such as, thepatient's facial features 1602, as well as the target 108 within itsfield of view 104, in order to register a relationship between thetarget 108 and the patient's head 1604. An optical image 302 generatedby the sensor 102 with the target 108 and facial features (the tip ofthe nose 1606 and the corner of the eye 1608, for example) within itsfield of view is illustrated in FIG. 16 b.

Registration of anatomical features (e.g. vertebrae, facial features) orother spatial attributes using an optical sensor may be more challengingthan measuring the pose of geometrically well-defined objects such assurgical tools. This is because the anatomical features are lessgeometrically well-defined. Furthermore, anatomical features may beoccluded by soft tissues, blood, etc., making them more difficult todetect using an optical sensor.

In one embodiment, the anatomical geometry is known a priori to thecomputing unit 110 for use in calculating a registration of theanatomical geometry. As illustrated in FIG. 17 , the anatomical geometrymay be in the form of a pre-operative scan 1702 (e.g. CT or MRI of theanatomy) that is specific to a patient. The known anatomical geometryaids in solving for the relative pose between the anatomy and thetarget, as it facilitates mapping the optical sensor image of theanatomy to the 3D scan 1702. The computing unit 110 relies on acorrespondence between anatomical features in the optical sensor image302 with corresponding anatomical features in the 3D scan 1702. Forexample, with reference to FIG. 16 a and FIG. 17 , correspondinganatomical features (e.g. the tip of the nose 1606, the corner of theeye 1608) may be identified on the 3D scan 1702 and on the optical image302. The correspondence of points on the optical sensor image 302 andthe 3D scan 1702 may be created automatically (i.e. by the computingunit), semi-automatically (i.e. by the computing unit with limited userintervention) or manually (i.e. by the user, for example, identifyingcorresponding points by clicking on them on a computer screen). Anynumber of corresponding points may be used to aid in the calculation ofthe registration. Rather than a 3D scan, a generic 3D model may be usedwhere the 3D model describes the anatomy with sufficient accuracy forthe purposes of registration e.g. using a 3D model of a face that is notspecific to the face of a patient undergoing surgery.

In another embodiment, the anatomical registration may rely on multiplemeasurements from a plurality of vantage points of the anatomicalfeatures and the target. FIG. 18 a and FIG. 18 b illustrate two opticalsensor images from two vantage points. Measurements from a plurality ofvantage points provide more geometrical information content to registerthe anatomy with high accuracy. It is particularly advantageous for themeasurements from different vantage points to include the target in theoptical sensor image since the pose of the target may be measured withhigh accuracy. Knowledge of the various vantage points (e.g. knowledgethat two vantage points are separated by a particular distance or angle)when detecting anatomical features that are part of the anatomicalregistration eliminates unknown variables when the computing unit solvesthe pose (i.e. the “extrinsic parameters” between sensor vantage pointsis known, and does not need to be solved for). A person skilled in theart will appreciate that having known extrinsic parameters for thecamera improves registration accuracy.

A user may receive feedback from the computing unit e.g. through adisplay unit, providing instructions to the user to capture multiplemeasurements from a variety of vantage points. For example, theinstructions may direct a user to capture a certain number of opticalsensor images containing the target and the anatomy to be registeredfrom prescribed viewing angles and vantage points. The images may bemeasured discretely or continuously (i.e. a continuous collection of aplurality of views).

In another embodiment, where the system is collecting a plurality ofimages of the target and anatomy, the system is tolerant of not beingable to identify any number of markers (i.e. identifiable features onthe target) for a subset of the images (e.g. because the markers areoccluded, outside the sensor field of view, etc.). This is achieved byinitially tracking both the target and the anatomical features, and whena subset of the markers are not available, tracking only based on theavailable target markers (if any) and the available anatomical features.

Using Other Scanning Modalities for NFS

In addition to identifying features (e.g. anatomical features) in anoptical sensor image for anatomical registration, it may be advantageousfor the sensor to measure the depth of objects within its field of view.A depth image from a depth image sensor with a field of view thatoverlaps with the field of view of the optical sensor may beadvantageous in calculating the pose of an object based on its naturalfeatures. In one embodiment, a depth image is provided to the computingunit by the sensor via a depth image sensor. Whereas an optical sensorimage is a measurement of light intensity of a scene on a plurality ofpixels, a depth image is a measurement of distance of objects in a sceneto the depth image sensor on a plurality of pixels. It is advantageousfor the depth image and optical sensor image to be overlapping whencapturing images of an object, since the optical and depth measurementscan be combined to aid in measuring pose of objects. Furthermore, depthimages of an object may be captured while tracking a target affixed tothe object, and used to reconstruct the 3D surface profile of theobject.

The optical image and the depth image are preferably related to a commoncoordinate frame (aka “co-registered”); such that both optical and depthmeasurements may be used to calculate the relative pose between a target(coupled to the anatomy) and the anatomy itself. The coordinate framesof the optical sensor and depth sensor may be co-registered by design,through factory calibration, etc. with a fixed and known positionalrelationship between the optical sensor and the depth sensor.

In one embodiment, the depth image sensor is a time-of-flight imager.The time-of-flight imager produces an optical sensor image and a depthimage using the same optical components (for e.g. the imager chip). Thisis advantageous as the depth and optical images overlap by default. Theoptical and depth images are inherently co-registered since they aregenerated by the same imager chip.

In one embodiment, the depth image is generated by the optical sensor incombination with modulated illumination. In its simplest form, a depthimage may be generated by applying a constant illumination to a scene,and measuring brightness of content within the scene; the brightness ofthe content is correlated to proximity (i.e. depth). Alternatively, theillumination may be modulated temporally (e.g. a sinusoidal wave ofillumination intensity) or spatially (e.g. illumination sequentiallyfiring from multiple points on the sensor). The illumination modulationmay be used to enhance the ability to accurately generate a depth image.Using modulated illumination is advantageous since it relies on theoptical sensor to obtain depth information and this configuration isinherently co-registered since a single sensor is being used.

In one embodiment, the sensor comprises components that projectstructured light onto the object being viewed by the optical sensor. Thestructured light is reflected back to the optical sensor and provided tothe computing unit. The computing unit may receive the optical sensorimage, and generate a depth image of a scene within which lies theobject, based on how the structured light appears in the optical sensorimage. This configuration is advantageous because it relies on theoptical sensor to measure/estimate depth; also, this configuration isinherently co-registered. This configuration requires that thecomponents that project structured light be co-registered to the opticalsensor.

In one embodiment, a laser scanner is coupled to the sensor, andconfigured to generate a depth image. Where the sensor is configured tomeasure the pose of a target (coupled to an object), and the laserscanner is configured to generate a depth image of the object, and wherethe laser scanner and the sensor have a co-registered relationship, thedepth image of the object (associated with the pose of the object) maybe known relative to the target. The laser scanner may be a separatedevice that is able to kinematically mate with the sensor in a knownco-registered position. The laser scanner may then be used duringanatomical registration, and removed otherwise, so as not to increasethe size/bulkiness of the sensor.

In the previously described embodiments, it may be advantageous tofurther utilize multiple views of the target and the anatomy to beregistered, from different vantage points. Depth image sensors may besusceptible to noise and inaccuracies; utilizing multiple views mayincrease the accuracy of the reconstructed depth map. Similar to apreviously-mentioned embodiment, having known extrinsic parametersbetween two optical/depth measurements of a scene may greatly improvethe accuracy, robustness and reduce the computational burden of posemeasurement of the anatomy.

In the previously described embodiments, it may be advantageous toutilize a 3D scan of the anatomy (e.g. CT or MRI scan) when solving forrelative pose between the target and the anatomy. Point correspondencemay be performed between the various locations on the 3D scan and thecorresponding locations on the optical sensor image and/or the depthimage. A generic 3D model of the anatomy may also be used, rather than aspecific 3D scan of the anatomy.

In one embodiment, a sensor is attached to a calibration probe of aknown geometry (e.g. known length). The calibration probe is intended tocalibrate the depth image sensor. When the calibration probe is broughtinto contact with an object, the object is at a known distance (depth)from the sensor. A depth image is measured at this time, and the depthimage is calibrated based on the known depth. Furthermore, thecalibration probe may be optically trackable via its natural features tofacilitate calibration between the optical sensor and the depth sensor.The computing unit in communication with the sensor may perform acalibration or verification based on the optical and depth images.

The above embodiments may be useful in a variety of surgicalapplications, including the previously-presented cranial application,wherein it is desirable to localize relative to a patient's head. Otherapplications are contemplated, such as Total Hip Arthroplasty. Forexample, the anatomy may be an acetabulum of a pelvis. It may bedesirable to know the registration between a target, attached to thepelvis, and the acetabulum. Alternatively, it may be desirable simply toknow the surface profile of a reamed acetabulum, for example, to checkwhether the acetabulum was reamed eccentrically. Should eccentricity (orother defects which lead to poor cup seating) be detected, correctiveaction may be taken by the surgeon prior to implantation of the cup. Anadvantage of this approach is to avoid removing and/or repositioning thecup. Cup removal may result in scrapping the original cup (since furtherreaming is required, increasing the size of the required cup).

FIG. 19 depicts a sensor 102, comprising an optical sensor and depthsensor, with an overlapping field of view 1901 within which lies atarget 108 attached to a pelvis and an exposed acetabulum. The sensor102 scans the surface of the acetabulum 1902 with light 1904 (e.g.structured light). The sensor 102 connects to a computing unit 110 thatcan perform further calculations to determine registration of theacetabulum to the target.

An alternative configuration is illustrated in FIG. 20 , wherein thesensor 102 is mounted to the anatomy (the pelvis), and the target 108 ismounted onto a surface scanner 2002 (comprising a depth sensor). In thisconfiguration, the target and the depth sensor have a co-registeredrelationship as the optical sensor can track the pose of the target thatis attached to the depth sensor. Furthermore, the depth sensor and theoptical sensor both communicate with the computing unit 110.

Hardware to Support Tracking Targets and Natural Features

A camera may be used to track natural features. The camera's spectralrange (i.e. the range of light frequencies to which it is sensitive)must be able to detect natural features. Typically, visible spectra aregood for this purpose; however, with appropriate illuminatingcomponents, infra-red or near infra-red spectra may also be suitable. Ifmarkers on the target are accurately and positively identifiable in thesame spectra as the natural features, then a fixed camera (including thelens, optical filtering, imager, and illuminating components) may beutilized.

Where different spectra are used for tracking of natural features andtracking of targets, a switchable optical filter may be utilized withinthe sensor. For example, in FIG. 21 , an optical filter 2102 iselectronically switchable via a solenoid assembly 2104 to switch betweenunfiltered and filtered light passing through to a lens 2106 on to animager 2108. Rather than a solenoid, a motor with a filter wheel may beused, and synchronized with the exposure cycles of the camera. Forexample, alternating image frames may be synchronized such that they arewith/without filtering. That is, for a 40 frames-per-second camera, itis possible to obtain two 20 frames-per-second video feeds in twodifferent spectra.

Where different spectra are used for tracking of natural features andtracking of targets, a spectrum splitter may be utilized within thecamera. As illustrated in FIG. 22 , a prism 2202 may be used for thispurpose. In this figure, two imagers 2204, 2206 are depicted, eachdedicated to a different spectral response of incoming light. There isno limitation to dividing the spectrum into two; more divisions of thespectra may be created as necessary.

Similarly, where different spectra are used for tracking of naturalfeatures and tracking of targets, a second optical sensor may beintegrated within the sensor to detect light in the other spectrum (i.e.the sensor is comprised of two optical sensors operating in twodifferent spectra). In this case, both cameras must be related to acommon coordinate frame, or co-registered (e.g. through manufacturing,factory calibration, etc.).

In embodiments where the sensor utilizes structured light for depthand/or pose measurement, the sensor may incorporate a structured lightprojector that projects structured light detectable by the opticalsensor into the optical sensor's field of view. In embodiments utilizingmodulated illumination for generation of depth images, the sensor mayincorporate the necessary digital and/or analog circuits to implementand/or control the modulation of illumination being projected into theoptical sensor's field of view.

Reference is now made to FIG. 23 . There is disclosed acomputer-implemented method 2300, comprising: at step 2302, a computingunit in communication with a sensor comprising an optical sensor,calculating a pose of a target in up to 6 DOF using optical measurementsgenerated by the sensor. At step 2304, the computing unit determines anexpected location of the effector of the tool based on pre-loadedinformation of the tool, the sensor being attached to the tool at aknown positional relationship. Step 2306 involves determining a locationof the effector of the tool based on features detected from an opticalsensor image generated by the optical sensor. At step 2308, a differenceis calculated between the location and expected location of the effectorof the tool, leading to the generation of a confidence metric using thedifference at step 2310.

Reference is now made to FIG. 24 . There is disclosed acomputer-implemented method 2400, comprising: at step 2402, a computingunit in communication with a sensor comprising an optical sensor,calculating a pose of a target in up to 6 DOF using optical measurementsgenerated by the sensor. At step 2404, the computing unit determines anexpected location of the effector of the tool based on pre-loadedinformation of the tool, the sensor being attached to the tool at aknown positional relationship. Step 2406 involves generating a virtualtool projection based on the expected location of the effector of thetool, and at step 2408, a composite image is generated by the computingunit comprising an optical sensor image generated by the optical sensorand the virtual tool projection, for display on a display unit.

Accordingly, it is to be understood that this subject matter is notlimited to particular embodiments described, and as such may vary. It isalso to be understood that the terminology used herein is for thepurpose of describing particular embodiments only, and is not intendedto be limiting.

As will be apparent to those of skill in the art upon reading thisdisclosure, each of the individual embodiments described and illustratedherein has discrete components and features which may be readilyseparated from or combined with the features of any of the other severalembodiments without departing from the teachings herein. Any recitedmethod can be carried out in the order of events recited or in any otherorder which is logically possible.

What is claimed is:
 1. A system to determine object poses during asurgical navigation procedure, the system comprising: a sensor unitcomprising an optical sensor configured to generate images of objectswithin a field of view of the optical sensor; and a computer processingunit, in communication with the sensor unit and memory storinginstructions that, when executed by the processing unit, cause thesystem to perform a method comprising: storing first natural featureinformation of a first object, the natural feature information defininginformation to identify the first object in the images using naturalvisual features of the first object; receiving from the sensor unit animage of the first object; extracting second natural feature informationfrom natural visual features of the first object in the image; andcalculating the pose of the first object relative to the sensor unitusing the first natural feature information and the second naturalfeature information to facilitate performance of the surgical navigationprocedure.
 2. The system of claim 1, wherein each of the first naturalfeature information and second natural feature information is definedfrom one or more of the natural visual features comprising inherentspatial features comprising any of a shape feature, a pattern feature, aconfiguration feature or an ornamentation feature.
 3. The system ofclaim 2, wherein the one or more of the natural visual features relateto portions of the first object without including any markers oridentifiers applied to the first object for purposes of tracking thefirst object.
 4. The system of claim 1, wherein the pose is calculatedin any number of degrees of freedom between 2 degrees of freedom and 6degrees of freedom.
 5. The system of claim 1, wherein the sensor unit isconfigured for attachment to a portion of a patient's anatomy.
 6. Thesystem of claim 5, wherein the portion is a pelvic bone.
 7. The systemof claim 1, wherein the first object is an acetabular impactor.
 8. Thesystem of claim 1, wherein the first natural feature information and thesecond natural feature information each comprises information definingedges of a cylindrical shaft of the first object.
 9. The system of claim8, wherein operations to calculate the pose account for a perspectiveeffect of the natural visual features in the image.
 10. The system ofclaim 1, wherein the instructions cause the system to simultaneouslylocalize targets and natural visual features from the image receivedfrom the sensor unit.
 11. The system of claim 10, wherein: the imagereceived includes the first object and a second object, the secondobject having a target coupled thereto; and the instructions cause thesystem to: localize the first object via the first natural featureinformation and the second natural feature information; and localize thesecond object from target information obtained from the image and storedtarget information for the target.
 12. The system of claim 11, whereinthe first object comprises a first instrument and the second object asecond instrument and wherein the instructions cause the system todetermine poses of each of the first instrument and second instrument.13. The system of claim 11, wherein: the first object is an acetabularimpactor; the second object is a patient leg; and the instructions causethe system to: localize the acetabular impactor relative to a patientpelvis; and localize a position of the patient leg relative to thepatient pelvis using a target attached to the leg.
 14. The system ofclaim 1, wherein: the sensor unit is configured to provide first imagesof the field of view in a first light spectrum and second images of thefield of view in a second light spectrum that is different from thefirst light spectrum; the image comprises a first image from the firstimages, the first image including a first object; and the instructionscause the system to: track the first object using the natural visualfeatures of the first object; receive a second image comprising one ofthe second images, the second image including a target coupled to asecond object; and track the second object using the target; wherein thefirst object and second object are tracked relative to a same coordinatereference of the sensor unit.
 15. The system of claim 14, wherein: thefirst object is an acetabular impactor and the first spectra is avisible light spectrum; the second object is a patient leg and thesecond spectrum is an infra-red spectrum; and the instructions cause thesystem to: localize the acetabular impactor relative to a patientpelvis; and localize a position of the patient leg relative to thepatient pelvis using a target attached to the leg.
 16. The system ofclaim 1, wherein the instructions cause the system to obtain the firstnatural feature information from a digital file accessible to theprocessing unit.
 17. The system of claim 16, wherein the first object isa tool comprising an effector, and the first natural feature informationincludes a positional relationship of the effector with respect to otherparts of the tool.
 18. The system of claim 1, wherein to calculate thepose the instructions cause the system to perform optimizationoperations to: match at least some of the first natural featureinformation and at least some of the second natural feature information;and calculate the pose responsive to the match.
 19. The system of claim1, wherein the instructions cause the system to compute the firstnatural feature information using a feature learning procedure in whichsufficient views of the object are presented to the processing unit forlearning its natural visual features.
 20. The system of claim 1, whereinthe sensor is configured with a light to illuminate the natural visualfeatures in the field of view.